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ABSTRACT 



Educational Testing Service, in cooperation with 
Response Analysis Corporation and MATHEMATICA, is carrying out a 
research project to describe reading behaviors of American adults and 
to translate these behaviors into a prototype of tasks for the 
measurement of adult reading achievement. A household interview 
survey of a nationally representative sample of adults will be 
carried out to collect data on existing reading behavior of all 
types. Then panels of prominent individuals representing major 
constituencies within the American public will discuss the benefit 
they feel would accrue to individuals who can successfully perform a 
wide variety of reading tasks. They will be provided with a summary 
of the survey results and a limited number of prototype reading 
tasks. The outcome of these meetings will be a picture of the judged 
benefit values of each reading task as it relates to the particular 
needs and aspirations of a wide variety of population subgroups. The 
completed prototype will be presented to respondents to receive data 
parallel to those obtained in the first survey. It should be possible 
to develop from such a prototype a larger, flexible multipurpose pool 
from which reading tasks may be selected to meet user requirements. 
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Targeted Research and Development Program in 



Reading--The Search for Criteria* 



Donald A. Trismen 
Division of Educational Studies 
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Princeton, N. J. 



The Targeted Research and Development Program in Reading implies by its 



very title a built-in emphasis on program evaluation. It is a targeted program. 



It has a goal or a set of goa ls--something to shoot at. Perhaps thi^ would 



not seem very remarkable, except for our long record of emphasis on the process. 



as opposed to the outcomes, of education. If an effort as massive as the Right 



to Read is to be Judged at least in part by the quality and quantity of its 



outcomes, then some rather careful attention ought to be, and is being naid 



to the definition of those intended outcomes. 



My purpose in this paper is to describe a research project currently being 
carried out by Educational Testing Service, in cooperation with Response Analysis 
Corporation and MATHEMATICA. This project has as its primary purpose the de- 
scription of the domain of reading behaviors of the adult American population 
and the translation of these reading behaviors into a prototype set of tasks or 
items for the measurement of adult reading achievement. This prototype set of 
tasks will constitute the exemplars of a larger body of reading tasks, which 
can then be combined in various ways to define operationally some of the goals 
of the Right to Read effort. 



^Prepared for a symposium, "Targeted Research and Development Program in Reading, 
Phase I, Right to Read Effort, USOE National Center for Educational Research 
and Development," held on February 5, 1971, at the Annual Meeting of the American 
Educational Research Association, in New York. 
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Two required characteristics of the prototype set of reading tasks will 
receive special attention. First, the tasks must realistically reflect the 
actual everyday reading behaviors of American adults. Where necessary, the 
usual constraints on reading achievement test items will be lifted. Thus we 
are not a priori limited to tasks suitable only for group administration. 
Neither are we restricted to a paper and pencil response mode. If the typical 
real world response to a reading situation is an oral or an action response, 
the development of a reading task patterned after the actual situation will be 
considered. Second, associated with each reading task will be information 
concerning the benefits which may be expected to accrue to individuals who are 
able successfully to perform that task. Of course, the concept of "benefit" 
is not unidimensional, and analyses are planned to define its several meanings 
more precisely. In general, the information about the benefit value of reading 
tasks will be of two kinds: self-reported demographic characteristics and 

perceptions of benefit by persons to whom the tasks are administered, and 
judgments of persons representing several important constituencies within the 
national population. 

At this point, I think it would be useful to give a brief overview of 
the major activities of the ETS research project, with minor excursions into 
some of the more interesting research issues. As an obvious first step toward 
the development of a pragmatically oriented set of reading tasks, it was con- 
sidered necessary to find out what American adults read. However, in order 
to provide sufficient guidance for the task development process, it was 
necessary to go far beyond a simple determination of "what." We found that 
we are equally interested in questions of "when," "why," "by whom," "how 
often," "under what circumstances," and "how important." Although information 
on reading habits exists, it was decided that only a special survey could 
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provide the particular array of answers which were needed. Therefore, the 
first major element of the project will be a household interview survey of a 
nationally representative sample of adults, tentatively scheduled for March 
and April of this year. Respondents will be asked about their reading activ- 
ities within four content categories— books , newspapers, magazines, and mail. 

In addition, they will be asked about reading associated with several classes 
of daily activity--mea Is , work, working around the house, school, traveling 
or commuting, shopping, club or church, theater, game, or other public event, 
and recreation and free time. The interviews are designed to elicit reports 
of reading behavior at all levels, from the technical journal to the corner 
stop sign. Respondents will be asked the duration of each reading activity 
their perception of its importance to them, whether they experienced any 
reading difficulty, and if so, why. In addition, a rather complete array of 
descriptive data will be obtained for each respondent, which will enable us 
to describe the degree to which he seems to be enjoying the economic, cultu,. il, 
and social benefits of our society. 

Although the interview procedure requires no reading by the respondent, a 
somewhat different body of Information will be obtained from self-reported 
non-readers. If the respondent cannot read English but is literate in another 
language, that language will be identified and a short summary of materials 
read will be obtained. If the respondent cannot read in any language, infonna- 
tion will be obtained concerning the kinds of materials which are read to 
him, and by whom. 

Data from the Reading Activities Survey will help to answer questions about 
rieadirg behaviors which (to exist. Although frequency, duration, and perceived 
importance of a given reading activity are important bits of information to 
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consider before including a task in the prototype set, they are not sufficient. 
In addition to knowing what Americ us do read, it was considered essenti il to 
obtain informed judgments of what they ought to read. In order to accomplish 
this, panels of prominent individuals who represent or can speak for major con- 
stituencies within the American public will be convened to discuss the kinds 
and degrees of benefit they feel would accrue to individuals who can success- 
fully perform a wide variety of reading tasks. Although no panelists have yet 
been selected, we are considering persons who might represent the point of view 
of such groups as education, labor, industry, research, the military, foreign 
language speaking minorities > and government. These panelists would be pro- 
vided with two kinds of information. First, a summary of the results of the 
Survey, so that they might have a vi Lid picture of what the American adult is 
in fact reading and what he thinks about it* Second, panelists will receive a 
limited nrmber of prototype reading tasks, which will serve as exemplars of the 
major classes of reading activities reported in the Survey. These prototype 
reading tasks will serve the critical function of bridging the gap between a 
verbal description of a reading activity and the operational definition of 
that activity-- that is, the reading task developed to measure it. 

The C’ttcome of these meetings will be a picture of the judged benefit 
values of each reading task as it relates to the particular needs and aspira- 
tions of a wide variety of population subgroups. There is no intention to 
iorce consensus upon these discussions, nor indded is there probably much hope 
of obtaining it. The primary aim is to assure the inc lusiveness of the proto- 
type set of reading tasks, rather than to restrict or balance it in arbitrary 
ways. Surely there must be some limitation upon its size, but the ultimate 
designation of tasks to be included will probably follow some sort of disjunct: 
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selection model. That is, a task might be selected either because it represents 
a frequently occurring reading activity, or a rarely occurring activity which 
was nevertheless judged highly important by Survey respondents., or a rarely 
occurring activity which was unimportant to Survey respondents but which panel 
members felt ought to be emphasized. Needless to say, the development of a 
decision rationale which will preserve the wide variety of judgments we are 
sure to receive, yet constrain the prototype set of tasks to a feasible number, 
is one of the more difficult but challenging research problems we must solve. 

Following the panel meetings, the remainder c£ the reading task generation 
activity will take place. When the final prototype set has been completed, the 
tasks will be presented to samples of respondents under two rather different ad- 
ministration designs. In one instance, all or at least a substantial portion of 
the entire set of tasks will be administered to each examinee. These administra- 
tions will take place at centrally located testing centers, and will of necessity 
be relatively lengthy. The data will enable us to study task intercorrelations 
and therefore the dimensionality of the prototype set. In addition, either all 
or a representative subset of the tasks will be administered to a national sample 
of adults in a second household survey. Naturally, only a small number of reading 
tasks can be presented to a single respondent under these conditions, and there- 
fore an item sampling administration design will be employed. That is, random 
subsets of reading task s will be presented to random subsamples of respondents. 

In addition to performance data on the various read-i*ig >-asks, the Reading 
Performance Survey will yield data on respondents 1 cha.v .eristics and per- 
ceptions of benefit parallel to those obtained in the fivnf survey of reading 
activities. The final product of the project will t:hus be a prototype set of 
reading tasks, each associated with some rather unique information. On the 
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more conventional side, task difficulty indices for a variety of respondent sub- 
groups will be available, based on a national sample. Also associated with each 
task, however, will be data not normally available to those who wish to assemble 
items into measuring instruments. Benefit data of two kinds will be produced. 
First, the benefit of being able adequately to perform the task, as perceived 
and reported by the respondent himself. And second, benefit as manifest by some 
of the tangible and quantifiable characteristics of the respondent and his place 
of residence. 

Thus it should be possible to develop from such a prototype set of reading 
tasks a more complete item pool of tasks "like these. 11 From this larger item pool, 
one could then assemble custom made tests appropriate for a variety of purposes 
and for a variety of respondent populations, defined by a wide range of demo- 
graphic variables. 

Perhaps the most important point to be made about the prototype set of items 
to be developed is that it will not be an instrument. It will not be the cri- 
terion against which a static and predetermined Right to Read effort will be 
measured for the next many years. It will not be a fixed target which will force 
the direction of our shots for the next decade. It will be rather a flexible, 
multipurpose item pool from which reading tasks may be selected to meet user re- 
quirements with respect to reading content, process, response mode, and diffi- 
culty for a wide variety of target populations. The tasks will be validated 
against several dimensions of empirically determined benefit. And each will 
have associated with it indices of perceived benefit (a concept not too far 
removed from something called "relevance"). 

Perhaps the thought has occurred to you that an infinitely variable target 
:Ls no target at all. One of my favorite cartoons depicts a servant with an 
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archery target strapped to his back, running madly in an effort to catch up 
with an in-flight arrow just fired by a determined looking king. We are not 
developing this kind of criterion either. The reading tasks to be developed 
will be those which someone--either experts, lay public, or both--has judged 
to be important and useful goals in our society. The flexibility at which we 
aim will operate with in these boundaries of empirical and face validity. There 
can be no quarrel with the potentially movable target, provided its position is 
selected before the arrow is fired. 
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